{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keras 循环神经网络\n",
    "递归神经网络（RNN）是一类神经网络，对于建模序列数据（例如时间序列或自然语言）非常有力。\n",
    "\n",
    "RNN层使用 for 循环在序列的时间步上进行迭代，同时保持内部状态，该状态对迄今为止已看到的时间步的信息进行编码。\n",
    "\n",
    "Keras RNN API 的设计注意点是：\n",
    "1. 易于使用：内置的 tf.keras.layers.RNN、tf.keras.layers.LSTM、tf.keras.layers.GRU 图层使你能够快速构建循环模型，而不必进行艰难的配置选择。\n",
    "2. 易于定制：你还可以通过自定义行为定义自己的 RNN 单元层（for循环的内部），并将其与通用tf.keras.layers.RNN 层（for循环本身）一起使用。这使你能够以最少的代码灵活地快速原型化不同的研究思路。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import collections\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "from tensorflow.keras import layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 创建一个简单模型\n",
    "Keras 中有三个内置的 RNN 层：\n",
    "1. [tf.keras.layers.SimpleRNN](https://tensorflow.google.cn/api_docs/python/tf/keras/layers/SimpleRNN)：一个完全连接的RNN，来自先前时间步的输出将馈送到下一个时间步。\n",
    "2. [tf.keras.layers.GRU](https://tensorflow.google.cn/api_docs/python/tf/keras/layers/GRU)最先在使用 RNN 编码器/解码器进行统计机器翻译的学习短语表示中提出。\n",
    "3. [tf.keras.layers.LSTM](https://tensorflow.google.cn/api_docs/python/tf/keras/layers/LSTM)：最早在长期短期记忆中提出。\n",
    "2015年初，Keras 拥有 LSTM 和 GRU 的第一个可重用的开源Python实现。 \n",
    "\n",
    "下例是一个顺序模型的简单示例，该模型处理整数序列，将每个整数嵌入到64维向量中，然后使用 LSTM 层处理向量序列。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding (Embedding)        (None, None, 64)          64000     \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (None, 128)               98816     \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 164,106\n",
      "Trainable params: 164,106\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = tf.keras.Sequential()\n",
    "# 增加一个期望输入为 vocab 1000的嵌入层，输出嵌入尺寸为64\n",
    "model.add(layers.Embedding(input_dim=1000, output_dim=64))\n",
    "\n",
    "# 添加一个包含128个内部单元的LSTM层\n",
    "model.add(layers.LSTM(128))\n",
    "\n",
    "# 添加一个 10 个单元的密基层\n",
    "model.add(layers.Dense(10))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 输出和状态\n",
    "默认情况下，RNN 层的输出每个样本包含一个向量。该向量是与最后一个时间步相对应的 RNN 单元输出，其中包含有关整个输入序列的信息。此输出的形状为（batch_size，units），其中 unit 对应于传递给图层构造函数的 units 参数。 如果你设置 return_sequences = True，则 RNN层 还可以返回每个样本的完整输出序列（每个样本每个时间步一个向量）。此输出的形状是（batch_size，timesteps，units）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, None, 64)          64000     \n",
      "_________________________________________________________________\n",
      "gru (GRU)                    (None, None, 256)         247296    \n",
      "_________________________________________________________________\n",
      "simple_rnn (SimpleRNN)       (None, 128)               49280     \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 361,866\n",
      "Trainable params: 361,866\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = tf.keras.Sequential()\n",
    "model.add(layers.Embedding(input_dim=1000, output_dim=64))\n",
    "\n",
    "# GRU的输出将是维度的 3D 张量（batch_size，timesteps，256）\n",
    "model.add(layers.GRU(256, return_sequences=True))\n",
    "\n",
    "# SimpleRNN 的输出将是维度的 2D 张量（batch_size，128）\n",
    "model.add(layers.SimpleRNN(128))\n",
    "\n",
    "model.add(layers.Dense(10))\n",
    "\n",
    "model.summary() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "另外，RNN层可以返回其最终的内部状态。返回的状态可用于稍后恢复 RNN 执行或初始化另一个 RNN。此设置通常在编码器——解码器逐序列模型中使用，其中编码器的最终状态用作解码器的初始状态。 要将 RNN图层配置为返回其内部状态，需要在创建图层时将 return_state=True。\n",
    "\n",
    "注意：LSTM 具有2个状态张量，但 GRU 仅具有1个。 要配置图层的初始状态，只需使用其他关键字参数initial_state 调用图层即可。\n",
    "\n",
    "注意：状态的形状需要与图层的单位大小匹配，如以下示例所示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_vocab = 1000\n",
    "decoder_vocab = 2000\n",
    "\n",
    "encoder_input = layers.Input(shape=(None, ))\n",
    "encoder_embedded = layers.Embedding(input_dim=encoder_vocab, output_dim=64)(encoder_input)\n",
    "\n",
    "# 状态返回不仅有输出\n",
    "output, state_h, state_c = layers.LSTM(\n",
    "    64, return_state=True, name='encoder')(encoder_embedded)\n",
    "encoder_state = [state_h, state_c]\n",
    "\n",
    "decoder_input = layers.Input(shape=(None, ))\n",
    "decoder_embedded = layers.Embedding(input_dim=decoder_vocab, output_dim=64)(decoder_input)\n",
    "\n",
    "# 将这两种状态作为初始状态传递到新的LSTM层\n",
    "decoder_output = layers.LSTM(\n",
    "    64, name='decoder')(decoder_embedded, initial_state=encoder_state)\n",
    "output = layers.Dense(10)(decoder_output)\n",
    "\n",
    "model = tf.keras.Model([encoder_input, decoder_input], output)\n",
    "model.summary()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6rc1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
